0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE
Computer Graphics and Applications
XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE
Narrative Physicalization: Supporting Interactive
Engagement with Personal Data
Maria Karyda
Department of Design
Aalto Universtiy
Espoo, Finland
maria.karyda@aalto.fi
Danielle Wilde
University of Southern Denmark
Kolding, Denmark
d@daniellewilde.com
Mette Gislev Kjærsgaard
University of Southern Denmark
Kolding, Denmark
mgk@sdu.dk
Physical engagement with data necessarily influences the
reflective process. However, the role of interactivity and narration
are often overlooked when designing and analyzing personal data
physicalisations. We introduce Narrative Physicalisations,
everyday objects modified to support nuanced self-reflection
through embodied engagement with personal data. Narrative
physicalisations borrow from narrative visualizations, storytelling
with graphs, and engagement with mundane artefacts from data-
objects. Our research uses a participatory approach to research-
through-design and includes two interdependent, studies. In the
first, personalized data physicalisations are developed for three
individuals. In the second, we conduct a parallel autobiographical
exploration of what constitutes personal data when using a Fitbit.
Our work expands the landscape of data physicalisation by
introducing narrative physicalisations. It suggests an experience-
centric view on data physicalisation where people engage
physically with their data in playful ways, making their body an
active agent during the reflective process.
Keywordsdata physicalisation, personal data, embodiment.
I. INTRODUCTION
Through physicalisation, digitalized personal data can
acquire physical properties [16], invite people to “feel their
data” [9], self-reflect and arrive at nuanced understandings
about their data and themselves. To date, most personal data
physicalisations are autobiographical (e.g. [1]). In such
examples, a developer collects data, and through an iterative
process, synthesizes and constructs their personal
physicalisation. Baumer et al. [2] argue that to achieve
nuanced reflection, data must be synthesized, not simply
encountered. The aforementioned process affords such
synthesis for the developer, but not the viewer. In current
examples of personal data physicalisation [1,16,19], data is
encoded mainly in static representations. Being static limits
the opportunity for self-reflection for the viewer, as it does not
allow for ongoing synthesis of information.
Our research examines the potential of full body
engagements, and narrative construction storytelling with
personal data physicalisations. We hypothesis these two
approaches may afford opportunities for people to synthesize
their personal data and thus achieve nuanced self-reflection.
To explore this idea, we developed Narrative Physicalisations,
modified everyday objects that are interactive and designed to
support self-reflection. Our concept builds upon the notion of
narrative visualizations [14] and data-objects [19]. Narrative
visualizations are traditionally used to support text in
journalism. They can facilitate narration of “data storiesand
are interactive. In complement, data-objects sit at the
intersection of industrial design and data physicalisation. They
resemble familiar artefacts, and can thus be contextually
situated. They benefit from the familiarity that people have
with everyday objects, yet are modified to incorporate
personal data.
In this article, we present two interdependent experimental
studies. The first, a participatory study, unfolds the
development of three customized, narrative physicalisations
of participant data. The second, an autobiographical study
[10], was designed to challenge preconceptions around what
constitutes personal data. Both studies use Fitbit devices to
collect the data. Our studies extend the existing corpus of data
physicalisation research by introducing the concept of
narrative physicalisations. As we demonstrate, through
engagement with narrative physicalisations people’s
reflective processes are enhanced. This enhancement arises as
the entire body is involved in deciding how personal data is
synthesized. As a result, the data itself, depending on the way
it is experienced, supports the narration of different stories.
Our research examines:
How designers and developers might create
physicalisations for dynamic bodily engagement with
data?
How such physicalisations might facilitate self-
reflection, and what the role of narration and
storytelling therein might be?
II. RELATED WORK
Data physicalisation is experienced through the body and
may become a ticket-to-talk in daily encounters facilitating
reflection through bodily interaction and speech encounters.
In the phenomenological view, as people become skilled
with objects, the tools they use to enact each skill becomes an
extension of their body. This phenomenon is what Clark and
Chalmers call the extended mind [3]. The long cane used by
a blind person provides a much discussed, and readily
accessible example [11]. Critically, the feeling that a tool
becomes an extension of the body is not consistent.
Heidegger [5], for example, argues that while nailing with a
hammer, the tool may disappear into the experience.
However, if the hammer breaks, it suddenly becomes an
object in the world. These phenomena demonstrate the
complexity of physical engagement with our surroundings.
Indeed, Sheets-Johnstone argues for the primacy of
movement in cognition [13] and Gallagher and Zahavi that
the body literally shapes the mind [4]. These
phenomenological understandings of human engagement
with the material world are integral to our research.
In data physicalisation, people experience data in different
ways. Barrass’s singing bowl [1] allows him to experience
his blood pressure data through sound. This experience may
Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.
0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE
Computer Graphics and Applications
trigger unforeseen reflections. The most common senses
activated in data physicalisations are the “felt experience”
combined with sight [6]. The DNA ring [12], for example,
physically represents a person’s DNA. If the surface of the
ring is touched, the wearer can feel distinct data points. The
ring thus uses touch to engage with data. The ring is designed
to be worn on a daily basis, in real-world settings, where the
data may become a trigger for reflective discussion.
Similarly, Zhu et al.’s data-objects [19] explore the potential
of incorporating personal activity data into everyday objects
e.g. a mug with embedded data that can be experienced
through touch. In these examples, data physicalisations go
beyond their role as reflective object to incorporate dual
meaning. The objects suggest reflection through physical
properties, and may also be used to facilitate daily tasks. Such
objects become an “extension” of the user’s body through
familiar use. The data-objects become a ticket-to-talk and can
trigger a reflective process. However, the affordances of these
objects that the data is fixed in time does not allow for
deepened experimentation with the data.
In data-driven presentations when objects are taken out of
context, they can become facilitators to help people better
understand complex data. For instance, Hans Rosling
1
uses
everyday objects to explain complex data to various
audiences. He enriches this process by including aspects of
narration, achieving storytelling through the interaction with
the objects. In such data-driven presentations the nature of the
objects is not altered. Rather, the way the presenter interacts
with the objects alters their meaning. Through such data-
objects, self-reflection can become part of everyday life, and
interactivity and storytelling enhance people’s reflective
process. These elements remain underexplored in personal
data physicalisations.
III. STUDY DESIGN
Our research examines ways of combining activity data
and reminiscence to develop personalized data
physicalisations as prompts for self-reflection. We used a
participatory approach to research-through-design (pRtD)
[18] in two interdependent studies, over 9 weeks (Figure 1).
Study-1 began with semi-structured interviews of three
participants with differing interests. A Fitbit device was then
given to each of the participants to wear, for five weeks.
1
Hans Rosling. 2014. TED: Hans rosling’s ted talks.(2014)
Throughout the data-gathering, we followed participants’
data via the Fitbit software interface. We did not make
contact with them during this time. Our aim was to identify
whether participants’ physiological data would make evident
their unique lifestyles. After five weeks, a generative session
was held with each participant. These sessions used
physicalisations to make sense of participants’ personal data
and to speculate on “future data”.
Study-2, conducted in parallel (Week 15), was an
experimental autobiographical exploration in which the first
author (M) and a colleague (C) shared a Fitbit device,
exchanging it at weekly intervals. At the moment of exchange
M and C collaboratively reflected on the previous week’s
data. After three such discussions, they held a generative
session to co-construct physicalisations to reflect on their
data. This session concluded Study-2 (Timeline: Figure 1).
Drawing on both studies, we developed a single
physicalisation for each of the three participants in Study-1.
These were modified everyday objects with which the
participants were intimately familiar. We hypothesized that
the familiarity of the objects and mastery of use would render
the proposed interaction intuitive, and allow participants to
focus on their data.
Critically, the purpose of Study-2 was to gain first-hand
insights into the use of Fitbit, better understand and
empathize with participants, experience first-hand
physicalisation of personal data, question and challenge the
role of the person’ in data generation and data
physicalisation, and thus enrich our engagement, as
researchers, with our Study-1 participants. As with any
autobiographical exploration, this process challenges
subjectivity and influence on a project [10]. However, as the
research was examining personalized interaction with
personal data, we determined the insights afforded would be
invaluable. Indeed, Study-2 enabled us to reflect and
iteratively test design ideasresponsive to, yet independent
of, Study-1as the research evolved. This research design
aligns with the pRtD approach. It enabled us to deepen our
exploration, while gathering physiological data; during
generative sessions; and while prototyping physicalisations.
It allowed room for reflection during the design process, as
we hermeneutically cycled through the emerging findings
Figure 1.The research timeline showing the overlap in the two studies.
Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.
0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE
Computer Graphics and Applications
interpreting and reinterpreting our understandings of what
personal data is, and how we might design for it.
To make the interdependencies between the two studies clear,
we present insights from each research activity at the end of
each section. We hope the documentation will allow other
researchers to reproduce our process. However, as with all
research through design, there is no expectation that others
following the same process would produce the same or even
a similar final artifact [20].
IV. RECRUITING PARTICIPANTS
Activity trackers serve a variety of industries from
commercial to pharmaceutical. That suggests a wide
spectrum of users, from patients to fitness enthusiasts and
others who are simply curious. We therefore sought
participants representing different industries and self-
tracking interests.
STUDY-1: We recruited three participants via personal and
professional contacts and social media, Ke, Ku and Mi. We
searched for people not actively self-tracking but with some
experience of wearables; and with different professional
backgrounds and interests.
STUDY-2: Involved the first author (M), who had no prior
experience with self-tracking; and C, a Fitbit user, recruited
through personal connections. C’s prior knowledge and
experience with wearables enabled us to track how the study
engaged previously held opinions about self-tracking.
We use pseudonyms for our participants to enhance the
personal tone in our narrative, as our work focuses on the
idiosyncratic nature of person-hood. Following, we
chronologically unfold our studies, alternating between them
where needed. This approach reflects the intertwined and
unfolding nature of the research activities.
V. STUDY-1: INTERVIEWS, PERSONAL DATA AND PRIVACY
Week 1, we conducted semi-structured interviews with Ke,
Ku and Mi. Our aim was to learn about their habits, interests,
important life experiences, motivation to participate in the
project, their future goals and views on what constitutes
personal data. We also aimed at establishing a context for
participant data collection which would then help us make-
sense of the gathered information during the project. The
three participants were in very different life moments. Ke, a
base player, had just graduated from a master’s degree and
was seeking employment; Ku, a computer programmer, was
struggling with demanding university courses; Mi, a retired
cancer caregiver, would be in India during the data gathering
period, receiving treatment to release tension.
During the interviews, we gained our participants
informed consent for having their Fitbit data collected and
observed for a month. Then, each participant was given a
Fitbit and shown how it works. From that point, for five
weeks, we followed their data without establishing contact.
We observed their data graphs online and made our own
connections between the data and the information shared
during the interviews.
VI. STUDY-2: REFLECTIVE DISCUSSIONS
As noted, (above and in Figure 1), in weeks 14, M and C
shared a Fitbit. C had experience using the Fitbit as a personal
tracker and embraced its transformation into a collective
device.
At the beginnings of weeks 24 we held shared reflection
sessions. During these sessions, we used the online graphs
provided by FitBit as reminders of what had happened on any
day. The graphs helped M and C narrate their individual
experiences as if they were pictures of the place and the time
where data was generated. e.g. M (pointing to the graph):
But do you know what happened [here]? I am going to show
you. Notably, we found that the Fitbit data was sometimes
misleading. On a day where M achieved a large number of
steps, the device “assumed” based on her performance that
she was doing well. Whereas, M reflected that she was in an
emotionally difficult space on that day; her steps reflected
anxiety and frustration. This information nicely illustrates
how context was vital in making sense of the data.
Numerical data offers new perspectives, which can
motivate a person to change their behavior [9]. We found
that the additional factor of connecting another person with
the data helped motivate C to achieve or change their goals.
C: “When I had to do 600 steps I just started playing with my
cat and ran just to give [M] 10.000 steps. When [the device]
vibrated I went to bed”. This example demonstrates how
motivation can shift from achieving 10,000 steps for personal
reasons into achieving them for the other person. What
motivated C the most to do the remaining 600 steps was that
M would observe the graphs the following day. Having to
discuss whether or not she had achieved her personal goals
proved to be a more powerful motivator for C, than setting
her own goals and accounting only to herself. The human
factor thus shaped the device’s impact and on occasion
enabled C to strengthen their motivation through the device.
This is further evidenced in M and C’s online chat: C: Since
you are tracking me, if I have [the device] only for one week
it is funny because it is both annoying and interesting. I have
been checking it more often because I want to know if I have
these 10,000 [steps] or not so you can see that I have made
it. And: “Sometimes I really considered to take a walk. The
synergy between the human factors and the device helped C
reflect on her data, as well as her relationship with the device
itself. This reflection was afforded through her engagements
with another person (M). C’s behavior spoke to an idea of
personal data that goes ‘beyond the personal’.
Several times, C and M identified which graphs belonged
to each other without looking at the dates. C’s data graphs
had a similar visual flow every day and were identified with
ease. In contrast, M’s graphs had a variety of patterns that
made it difficult to identify what happened, and during which
day. As C explains to M: [your graphs] look like fifteen
different people.” That indicated that there is a distinct
‘personality’ to the data, even if we need the context to make
sense of them and even if they are interpersonally generated.
A. Insights from the reflective discussions
The reflective discussions above, demonstrate that the
graph formats can constitute tickets-to-talk about the lived
experiences behind the data. Further, the graphs can be used
Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.
0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE
Computer Graphics and Applications
to support people’s memory. Through reflective discussions
we learn about Cs data, as well her personal life. These
findings indicate that graphs can support narration, essential
for understanding the data. We explore this possibility in
Study-1: Our Study-1 participants’ lifestyles were very
different from each other. These differences were evident in
their graphs. After two weeks of observations it was possible
to identify which graphs belonged to which participant by
sight alone. At times, we saw nuances in the patterns, but they
were still recognizable. We determined that these nuances, in
combination with the graphs, constituted tickets-to-talk, and
used them to design a customized generative session with
each participant. The aim was to learn more about their
activities during the data gathering period.
VII. GENERATIVE SESSIONS
In week 5, we conducted individual generative sessions
with our Study-1 participants to scaffold qualitative
information about their lived experiences during the data
gathering period. Following, we held a generative session in
Study-2, to understand how to leverage the insights from such
sessions when developing data physicalisations. We begin
with Study-1.
A. STUDY-1: Generative Sessions with Ke, Ku and Mi
Each generative session was divided in three parts: i)
we conducted interviews seeking to understand the
participant’s personal experience with the Fitbit, and their
opinions about being part of the project; ii) their graphs were
collated and used to support the participants to recall
experiences, which were discussed in detail; iii) participants
were introduced to two interactive props for physicalizing
data (Figure 2a). They were invited to choose one and, based
on memory, physicalize an average daily graph of the past
month, then speculate on a personal future graph and try to
physicalize that. We present our findings:
In the first task, we asked participants about their
experience of using the Fitbit device. Ke and Mi explained
that their behavior was influenced by the device. In contrast,
Ku reported that the device did not motivate him to be more
active. Ke referred to a moment where he had completed
“9000 something steps” and to achieve his daily target of
10000, started walking around his room. Ku said he only
looked at his daily data following a stressful experience, as at
these moments he was curious about his performance. Ku:
“It was after the fact! I was checking the result from the day
before” ... “For different instances I would check different
things. If it was a stressful day, I would check the pulse.”
Noticeably, while Ku said he was not influenced by the
device, he also said that his goal was to provide us with
“real” data. Ku: “I wanted you to get the actual data of how
I live but it was also a conscious decision not to change it.
Notably, during the data gathering period Ku reduced his
alcohol consumption so that we would get ‘the actual data’.
Discussing a big night out, he explained: “I feel guilty
because of this all night out. From his discussions, we could
see that he was influenced and motivated by the data in a
similar way to C. The need to share his lifestyle with us,
influenced his actions.
In terms of motivation, Mi explains: “It did make me walk
more but not as much as if I had been at home.” What was
obvious in our interview with Mi was that the data she
produced in India did not correspond to how her data would
have been if she was in Denmark. This topic was brought up
repeatedly throughout the generative session.
During the second task, our participants selected one or
two days and annotated their graphs with what they could
recall and discussed their lived experiences. In Mi’s case, the
main topic was the different lifestyle she was introduced to
while being in India compared to when she is in Denmark.
Mi: “What I found interesting was the heart rate going down
while I was in India.” Mi’s resting heartbeat graph illustrated
that her heart rate average was lower while abroad. This
outcome, justified to her the effectiveness of her treatment in
India. Mi: “This is not the real me” she said, suggesting that
the data we were following was not a realistic data picture of
herself, as it corresponded to her wellness retreat in India,
where she was advised to reduce her walking for her
treatment.
The most significant event for Ke during the data gathering
period was receiving two employment offers on one day. He
could recall and describe this day in great detail. His narration
included receiving the job offers, sharing this information
Figure 2. Generative session, task 3: (a) presenting the two
probes that assist people to physicalize their data; (b) Ku interacting
with one of the two probes.
Figure 3. Three participants during the generative sessions.
Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.
0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE
Computer Graphics and Applications
with his family and celebrating with friends. During the
celebration he consumed a lot of alcohol which resulted in a
high heart rate. Fitbit perceived this as exercise and
congratulated Ke with a virtual reward. Only when Ke
contextualized the data collection alcohol consumption
were we able to make-sense of his personal data.
The significant moment for Ku was a live performance
with his band. Before and after the concert his heart rate was
very high. He marked on the graphs when his heart rate was
high due to stress and when due to excitement. This again
surfaced a contradiction between the data graphs and the
lived experiences. Both heartbeat rates before and after the
concert were high. However, one peak corresponded to a
stressful moment, the other, with excitement or pleasure.
In the third part of the generative session (Figure 3), the
graphs that Ke and Mi physicalized were very similar to the
visualizations of Fitbit. In contrast, Ku (Figure 2b) focused
on narrating a story rather than creating a physicalisation that
would resonate with his graphs. The physicalisation he
developed helped him to share his story which again brings
the focus on aspects of narration and storytelling when
interacting with the graphs.
The most important finding of these generative sessions
was the ways that the experiences of participants linked to
their personal data. The interviews supported our premise that
data may partly reveal insights but is not able to demonstrate
the complexity and richness of an experience.
B. STUDY 2: Generative session M and C
The generative session with M and C served to explore
opportunities that data physicalisation may offer the
development of physicalisations for Study-1. The session was
divided into three parts. The first investigated the potential of
undated data cards (graphs) as reminders of specific events.
The second and third examined ways of co-creating
physicalisations based on those graphs.
In part 1, C was presented with three different graphs.
These graphs corresponded to the days of the month that the
shared Fitbit was handed over, and thus combined the data
from both C and M. The graphs were not dated; C was asked
to guess which days of the month were depicted on the cards.
We aimed to discover: i) if C was able to identify her personal
data; and ii) if she was able to recall a particular day simply
by looking at a graph. Notably, she characterized the data that
did not belong to her as faulty as she could not relate her
lifestyle to it.
Part 2 involved M and C working together to physicalize
the graphs discussed in part 1. The aim was to better
understand the role of materials when physicalizing a graph.
The first physicalisation was constructed from two pieces of
wood, each with 12 holes, and two roles of white and red
thread: one color for each person’s data-set (Figure 4b). The
artefact was painted white to represent a blank canvas. To
begin this exercise, C was asked to associate values to each
component of the physicalisation. She associated the top rod
to steps, explaining that the holes represented different
amounts of steps; and the bottom rod to time, and the shafts
to hours of the day. The white thread represented C’s data,
The red, M’s. The sole purpose of the different color threads
was to distinguish between the two datasets. The resulting
physicalisation shows one side in red and the other in white;
one side represents data from the morning, the other from the
evening (See Figure 5).
In part 3, C was asked to associate values to four differently
colored water glasses. Syringes were provided to use as a pen,
to ‘draw’ in color on a large paper sheet. The paper was
intentionally chosen such that the colored water would bleed,
and be somewhat uncontrollable. C associated blue with bad
feelings, red with good; black with time; and yellow with
activity (See Figure 4c).
At the end of the session, M and C analyzed the two
physicalisations. C proclaimed the first too abstract, as she
was not able to understand or read the graphs. To engage
reflectively with such an abstract physicalisation, a person
must learn to read it. The lack of specificity in the data and
the absence of what C felt were relatable material choices
seemed to eliminate any sense of the personal. C described
the resulting physicalisation as ’distant’. For the second
physicalisation, C explained that the way the color spread
provoked reflections about the nature of emotions and their
indeterminate form (Figure 4c). C: “The fact that it flows on
its own direction gives you the feeling that ok I can drop some
in many places so it makes it more real.” In this case, that
the physicalisation was abstract, less ‘precise, open for
interpretation, is what made it more ‘real’ for C. It allowed
for complexity and multiple interpretations of data. It was
Figure 4. l-r: a) laser cut activity graphs b) two-rod physicalisation c) physicalisation co-constructed by M and C using liquid colors.
Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.
0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE
Computer Graphics and Applications
deemed to come closer to ‘reality’ than a ‘fixed’, precise and
final-seeming form.
Because of these attributes, the second physicalisation
became a tool for reflection. It presented alternative image of
the participant’s self and prompted her to explore. New
relations between emotions and data, reality and extraction of
reality, were demonstrated here. This activity showed us that
it was possible to physicalize personal data in ways that
would add some of the richness of lived experiences.
Critically, the level of abstraction in the first physicalisation
seemed to depart too far from C’s experience. Whereas the
abstraction in the secondcreated by C, herselfsupported
the reflective process.
Overall, the autobiographical exploration showed how a)
visualizations of personal data can support remembering, and
b) for reflective physicalisations the format should support
subjective expression.
The choice of the forms and materials used in the
generative session were not as rich as previous studies where
a variety of materials are provided for people to engage in
open ended ways in prototyping 7,16]. However, similar to
[7], our design constraints afforded more “focused
engagement with physicalization construction. We
anticipated that a wider range of materials would make the
construction process more laborious. The two
physicalisations included abstract materials with diverse
properties. The aim was to cover different form and material
behaviors, and afford nuanced ideas and reflections. For
example, the physicalisation with two rods is a closed shape
which allows for limited extrapolation, while in the second
physicalisation, the colored water bleeds in ways that are
unpredictable affording nuanced reflections and insights.
VIII. STUDY 2: NARRATIVE PHYSICALISATIONS
To develop personalized physicalisations for Study-1, we
scheduled a two-week ideation phase. We reflected on the
themes that arose in the two studies and considered how to
highlight those themes. The strongest theme was that data is
always situated within a particular type of experience.
Two other criteria came out of the ideation phase: the
physicalisations i) should support data expression
through ambiguous formats, ii) should lay the ground
from which to recall lived experiences.
Based on these criteria, a personalized physicalisation was
developed for each participant. The participants’ character
traits also served as inspiration. We developed a foosball
table for Ke, a one string instrument for Ku and a treadmill
for Mi. Each of these included the data graphs in a unique
way (Figure 5). We invited our participants to interact with
these modified, interactive data objects. During the
interaction we asked open questions such as: “How did you
feel interacting with the object? Can you demonstrate how
did you feel that day?” (the day that the data was captured).
This approach aligns with Sanders and Stappers [15] work on
generative sessions. Following present the physicalisation
and testing.
A. The Foosball Table
Ke is a problem solver and a competitive person, who loves
foosball. We converged these traits to make him a personal
data Fusball Table (Figure 6). Our aim was to provoke him to
compete against his own data. The physicalisation had the
format of an ordinary foosball table with small modifications.
The handles of one opposite side were removed, so it could
only be played by one player. The remaining four handles
were made of wood and had a different number of foosmen
on each, so their weight varied. Handling the rods thus
required a specific learning of force and speed. A ping pong
ball was used to increase the level of difficulty and two fans
were placed closed to the net with the aim of blocking the
participant from scoring a goal the fans blow the ball away
from the target and so make scoring more challenging.
The end of the table was left open to allow a data-sheet to
rotate freely on the table’s surface, like a conveyor belt
(Figure 6). The data-sheet showed the heartbeat of three
graphs: one from the day that Ke received two employment
offers, one from the day before, and one from the day after.
These graphs were chosen to evidence his behavior change
on the day he received the employment offers. The rules of
the game were: a) keep the ball within the borders of the
paper, b) do not let the ball fall out of the foosball table, c)
Figure 5. The completed two-rod probe (study 2).
Figure 6. The Foosball table.
Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.
0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE
Computer Graphics and Applications
score by the end of each day (meaning the period of time that
a graph would need to start appearing from the net side until
the moment that it would disappear completely from the
surface of the table).
At first, Ke identified the differences and similarities of the
physicalisation with a traditional foosball table, saying that
the game was different but the elements were the same. The
process of learning how to use it and then score took a while,
as the physicalisation had different affordances than a regular
foosball table. Ke first interacted with the artefact without
activating the rotation of the paper. Once the rotation was
activated, Ke was asked to follow the rules.
We observed that the graph did not seem to interfere with
the experience. Ke later demonstrated how he would play one
of the visualized days. The graphs did not assist him to
remember or tell his story; rather, his narration depended
completely on his memories. Though Ke did not use the
graphs in the physicalisation, the rotating paper added a level
of difficulty to the experience. “It is annoying that the paper
is moving because I can’t rest.” His experience demonstrates
the asymmetrical relationship between the embodied
familiarity Ke has with a regular foosball table compared to
his interaction with the physicalisation. The only moment the
graph played a prominent role was when Ke attempted to
follow the line of the visualization. This action took the
experience far from the initial concept. At this moment, Ke’s
vocabulary was influenced by his actions. Even his speech
was shaped by and adjusted to what he was doing. It was
kind of all over the table he said, referring to his mood,
instead of saying all over the place.
While interacting with the foosball, Ke’s sense of the data
passed into the background. The physicalisation failed to
trigger competition. Nonetheless, it led to a rich discussion
with Ke. We suggested making the visualizations more
prominent in the interaction and asked him to speculate how
he might interact with that physicalisation. Ke responded:
The same agent is doing all these but has different wishes
meaning that the person who generates the data also wants
the data to be easy to play to then win the game. The same
person could also change his lifestyle to win but with the cost
of making life compromises with a risk of a positive or
negative outcome. Reflecting on the use of the
physicalisation Ke said: “I am creating the difficulty while
living my life”.
The physicalisation is not trying to digest the ‘data’ for Ke.
Rather it seeks to provoke reflection and critique by showing
how our engagement with data is not ‘neutral’. Data may
reflect an experience (or aspects of it) and may spark stories
and reflections upon that experience. Ke’s last remark
indicates that physical interaction with data has the potential
to change not only the experience of the data, but the person
engaging with it.
B. The One String Instrument
Our second participant, Ku, has contradictory traits in his
personality he is at once an introvert and a very social
individual. Ku was initially against the use of the tracking
device. We therefore aimed to develop a physicalisation that
would be pleasing for him to interact with. Fortunately,
finding what he was passionate about was easy as he is a
musician, a base player.
After working through several design ideas, we created a
one-string instrument (Figure 7). This design decision made
our design less complex than a multi-stringed instrument, yet
still engaging. The physicalisation consisted of one guitar
string, slightly elevated from the rest of its components. A
data-sheet of heartbeat visualizations slid under the string.
The heartbeat visualization was of four days during which Ku
was rehearsing and then performing in a concert. We
positioned the string against his heartbeat visualizations to
create a sonic asymmetry that made it difficult to generate a
pleasing sound. A plectrum and a glass container with a
cylindrical form were included to compliment the
physicalisation. Ku was asked to pluck the string with his
right hand and follow the line of the heartbeat data with the
glass container while the data-sheet slid under the string. He
thus could generate the soundtrack from each of his days.
Ke’s physicalisation is not dissimilar to a slide guitar and
was perceived as such. Ku has experience playing slide
guitar. This experience reinforced his excitement about
interacting with the physicalisation. His previous (embodied)
knowledge of playing string instruments helped him interpret
the graphs quickly. The way that the string was plucked in
parallel to the narration of his experiences seemed effortless.
Without being asked, Ku started following the line of the
graphs with the glass container, while plucking the string.
Unlike the foosball physicalisation, the data-sheet in this case
rotated from the beginning of the experiment. Ku: The first
thing I want to do is to follow the line. He described the
sound experience as almost disturbing or “messy. After a
while, he developed an understanding of the sounds related
to high or low peaks. He explained, because he is a bass
player, he enjoys lower sounds. He was enthusiastic about the
lower peeks relating to him being calm.
Figure 7. The one-string instrument.
Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.
0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE
Computer Graphics and Applications
As with Ke and the foosball physicalisation, Ku was asked
to imagine and freely play the sounds of the days that were
on the data-sheet, based on his lived experiences. In
particular, we asked: Can you play how those days felt?”
Here, a shift occurred in his interaction with the artefact.
Plucking the string and giving a high or low tempo according
to each day became more important to him than controlling
the sounds with the glass container. At a certain point, he
perceived the four graphs as a singular sonic experience. He
connected the high tempo to the excitement or the stress of
the days in question.
Similar to Ke’s experience with the foosball table, Ku’s
vocabulary was influenced by his interaction with the
physicalisation. He used the high tempo to describe the
intense days or, as he described them: days with pleasant or
unpleasant stress. Similar to C, his interaction with his
personalized physicalisation prompted him to share more
about the days of the graphs. This contrasted with his
reluctance to share during the generative session.
Ke’s physicalisation was deemed to have a personality’
that provided context for making sense of the personal data.
Similar to Ku’s experience with his personalized data
Foosball Table, Ku’s sense-making was described in relation
to the interaction with the data, rather than with the data itself.
C. The Treadmill
While in India on her stress-relief retreat, Mi, our third
participant was instructed to reduce walkingone of her
favorite activities. This compromise transformed her data
image. It presented a strong contradiction between her
impression of her data in Denmark versus in India, and
inspired her personalized physicalisation. During ideation,
we focused on Mi’s step visualizations. When walking one
proceeds by advancing the feet alternatively so there is
always one foot on the ground. Steps unfold in a corrugated
way. When juxtaposed to graphs of steps, the way the action
unfolds in the body does not seem to match the graphs. The
differences between the visual output of a graph and the way
the action of walking looks, were significant elements of our
ideation process. This difference led us to wonder how one
might walk if one needed to follow the image of a graph.
To make the physicalisation, we modified a treadmill
(Figure 8) to fit the design idea. The top part was transformed
into a speech roster, including a stand on which the graphs
could lie. Six non-consecutive graphs were chosen for this
activity. Two, from the days with the lowest and highest
values of steps. The others were discussed during the
generative session. All six were placed on the platform in a
specific order, building from the day that Mi walked the least
steps to the day with the most steps. A square acrylic sheet
with the same dimensions as the graphs was placed next to
them. The sheet showed a color scale of eight colors from
dark blue to light pink. It was to be placed on the graphs and
divide them into the eight differed colors. At the bottom of
the treadmill was a wooden square, hiding a projector. The
projector was connected to the treadmill. As Mi walks on the
treadmill she generates a projection of the color scale on a
large wooden canvas on the opposite was. The colors change
from dark blue to light pink, according to Mi’s walking speed.
As she walks, Mi recalls and narrates her experiences.
The physicalisation was controlled by Mi, without sup-port
of an external power supply. The treadmill thus required extra
effort. Similar, to the other physicalisations, to interact
required learning how to use the device. While walking on
the treadmill, Mi said: “I had to use more force than what I
was expecting.” Her speed correlated with her ability to
identify the colors. On her second attempt, Mi tried to go
through the entire color scale; every time a color appeared,
she walked more slowly. Ultimately, she was able to identify
the changes between the colors, leading her to reflect: “I was
thinking that I often walk quite fast, and I like it, but I also
really like the slow. It is like you see more the changes.” This
reflection was facilitated by the affordances of the prototype.
After Mi learned how to use the treadmill, she focused on
looking at the graphs. She interpreted a large number of steps
as high speed and a small number of steps as low speed. She
explained: “When I look at this, is life in slow” meaning that
her experience in India shifted her actions from very active to
a slower pace. From then on, when describing experiences
during the testing she refers to them as slow or fast,
influenced by the interaction with the physicalisation. Then
her discussion transitioned to other life experiences. Her
reflections triggered questions such as, “How would you walk
your first wedding” or “How would you walk the birth of your
first child.” Walking was related to feelings. Her speed
reflected the overall feeling she had of an experience.
Occasionally when the feeling of the experience in the past
did not match the feeling of the experience in the present, it
was difficult for her to adjust her speed to the experience. The
conflict in her mind was evident in her bodily movements.
IX. DISCUSSION
All three physicalisations in Study-1 provided the
participants with a double opportunity. First to interact with
their personal data and relate to it through embodied learning
Figure 8. The treadmill.
Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.
0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE
Computer Graphics and Applications
(Figure 9). Second, through this learning, and by associating
new identities to their data, they were able to explore,
understand, share and reflect on their experiences, and
personal data more broadly. The overall interaction may be
divided in two parts:
A. Read and Act
The first type of interaction we observed was being guided
by the graphs but ignoring which day they represented. This
interaction positions the visualizations as the main actors of
the activity. The person interacts with their data yet does not
connect the data to an experience. Despite the gap between
data and experience, the interactions trigger reflections about
personal data on a different level. For instance, for Ku, the
lower sounds resonated with him being calm, this evoked
good feelings about the data. This experience was not
connected to any real-life experience captured by the data.
The associated interaction triggered a kind of private
knowledge. For both Ku and Mi, we observed the data
determine and lead the interaction as it became the element
of attention.
B. Interpret and Act
In this second phase of the interaction, each person had
completed reading their graphs and was trying to recall their
experiences based on the days depicted. Critically, the
physicalisations are customized artefacts that our participants
were intimately familiar with. The interactions, thus, benefit
from participants’ embodied knowledge. While interacting
with their physicalisations, the participants associate feelings
with their personal data. The graph seemed to provide an
“impression of data, to play the role of a reminder. At times,
the personal data became transparent. For example, when the
participant focused on the interpretation of the experience
rather than on the data itself. The physicalisation thus became
a mediator through which the participants expressed their
feelings both bodily and through speech.
When the interpret and act interaction takes place, the
physicalisations become artefacts again in the world. They
become ready-to-hand tools that the participants know how
to use. The use of physicalisations in this way, leads to
reflections about people’s lived experiences. If we consider
that the participant’s body at once knows how to use an
artefact and is learning its use. The physicalisations thus have
a dual nature. They become the medium for interaction as
both master and student. The first nature is associated with
kinesthetic learning, the second focuses on emotions related
to lived experiences. Verbeek [17] argues that the mediating
role of technologies suggests two ways that artefacts can
influence humans being: as mediators of perception or
mediators of action. Artefacts become mediators of
perception when they mediate human experience and
interpretation of reality. e.g. technologies such as the
ultrasound shape the way we perceive and interpret a fetus
before it comes to the world. Mediation of action is related to
how artefacts incorporate meanings that provoke human
actions, e.g. a traffic sign signifies that drivers should slow
down, mediating this action through its material existence.
Similarly, data visualizations are mediators that do not
require an embodied relationship to provoke action. Instead,
they suggest action. Whereas, in personalized data
physicalisations, the graphs become ready-to-hand. They
transform into mediators of perception as participants, while
interacting with them, mentally build an interpretation of an
experience.
X. ON THE DEFINITION OF DATA PHYSICALISATION
Jansen et. al. propose that, “A data physicalization is a
physical artifact whose geometry or material properties
encode data.”
Our physicalisations do not explicitly follow this definition,
as they incorporate people’s visualizations. The graphs were
included as necessary elements for people to recall the
experiences and reflect, highlighting the important of context
and experience to spark reflection. Our work extends
understandings of personal data physicalisation by
introducing interactivity; by illustrating the person-data
relation as dynamic, complex, ambiguous and unsettled -
reflected even when data remains ‘graphical’; and by
affording interaction with data, rather than simply
physicalisation of data.
Critically, in our personalized data physicalisations, the
graphs acted as triggers for interaction with the artefacts; by
extension with the data; and thereby narrative offerings from
the participants. Hogan et al. [6] argue that in dynamic
physicalisations the data is encoded in the shape as well as
the behavior of the representation.” We extend that by
arguing that through bodily interactions with narrative
(interactive) physicalisations, the understanding of personal
data is extended. This extension allows the person to view
their personal human-data relationship as a whole rather than
as two separate elements. The data is encoded in the
interaction with the body and is extended by it.
XI. CONCLUSION
We presented a two-part study that resulted in three
personalized data physicalisations, what we call Narrative
Physicalisations. Our research demonstrates the potential in
narrative physicalisations to enable people to develop
meaningful narratives around data and personal experience.
Building on research into data objects, our findings
demonstrate that narrative physicalisations effectively
leverage the affordances of objects and the fact that people
know how to use objects. Following this work, designers and
developers of data physicalisations can use familiar objects
Figure 9. Ke and Ku interacting with the physicalisations.
Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.
0272-1716 (c) 2020 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.
This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/MCG.2020.3025078, IEEE
Computer Graphics and Applications
to leverage participants’ embodied knowledge, and prompt
them to reveal insights that may not otherwise come to mind.
Our focus has been on people’s ability to engage with their
personalized objects and, through that engagement, recall
personal stories and reflect upon person-data relationships.
For future research, we anticipate exploring other sense
modalities for personalized, kinesthetic, tasty, smelly,
stimulating narrative physicalisations based on personally
meaningful data.
REFERENCES
[1] S. Barrass. Acoustic sonification of bloodpressure in the form of a
singing bowl. In Conference on Sonification of Health and
Environmental Data (SoniHED), 2014.
[2] E. P.S. Baumer, V. Khovanskaya, M. Matthews, L. Reynolds, V. S.
Sosik, G. Gay. Reviewing Reflection: On the Use of Reflection in
Interactive System Design. In Proceedings of the 2014 Conference on
Designing Interactive Systems, 2014, pp.93102.
[3] A. Clark and D. Chalmers. 1998. The extended mind. analysis58, 1
(1998), pp. 719.
[4] Shaun Gallagher and Dan Zahavi. The phenomenological mind.
Routledge. 2012.
[5] M. Heidegger, J. Macquarrie, E. Robinson. Being and time, 1962.
[6] T. Hogan and E. Hornecker. Towards a design space for multisensory
data representation, Interacting with Computers, 2017, pp. 147-167.
[7] S. Huron, P. Gourlet, U. Hinrichs, T. Hogan, Y. Jansen. Let’s Get
Physical: Promoting Data Physicalization in Workshop Formats, In
Proceedings of the 2017 Conference on Designing Interactive Systems
(DIS ’17), New York, 2017, pp. 14091422.
[8] Y. Jansen, P. Dragicevic, P. Isenberg, J. Alexander, A. Karnik, J.
Kildal, S. Subramanian, K.Hornbæk. Opportunities and challenges for
data physicalization, In Proceedings of the 33rd Annual ACM
Conference on Human Factors in Computing Systems, 2015, pp. 3227
3236.
[9] D. Lupton. Feeling your data: Touch and making sense of personal
digital data, New Media &Society19, 2017, pp. 15991614.
[10] C. Neustaedter and P. Sengers. Autobiographical design in HCI
research: designing and learning through use-it-yourself, 2012, pp.
514523.
[11] M. Merleau-Ponty, Phenomenology of perception. London: Routledge,
(19452012).
[12] A. Rezaeian and J. Donovan. Design of a tangible data visualization.
In Proceedings of the 7
th
International Symposium on Visual
InformationCommunication and Interaction, 2014, pp.232235.
[13] M. Sheets-Johnstone, The Primacy of Movement, John Benjamins,
1999.
[14] E. Segel and J. Heer. Narrative visualization: Telling stories with
data.IEEE transactions on visualization and computer graphics16,
2010, pp. 11391148.
[15] P.J. Stappers and E.BN Sanders.Generative tools for context mapping:
tuning the tools.InDesign and Emotion. Taylor & Francis London,
2003, pp.7781.
[16] A. Thudt, U. Hinrichs, S. Huron, S. Carpendale. Self-Reflection and
Personal Physicalization Construction. In Proceedings of the 2018
CHI Conference on Human Factors in Computing Systems, 2018, pp.
113.
[17] P. Verbeek. Materializing morality:Design ethics and technological
mediation, Science,Technology, & Human Values31, 2006, pp. 361
380.
[18] Wilde, D. Designing Research Education for Global Challenges. She
Ji Journal of Design, Economics and Innovation. Vol. 6, Issue 2
Summer 2020, pp. 170-212
[19] C. L. Zhu, H. Agrawal, P. Maes. Data-objects: Re-designing everyday
objects astactile affective interfaces, InternationalConference on
Affective Computing and IntelligentInteraction, 2015, pp. 322326.
[20] J. Zimmerman, J. Forlizzi.Research Through Design in HCI. In: Olson
J., Kellogg W. (eds) Ways of Knowing in HCI. Springer, 2014,
Mary Karyda is a Doctoral Candidate in the Department
of Design at Aalto University, Helsinki. In her doctoral
research, Mary, explores meaningfulness - a part of
meaning making that relates to feelings - in relation to
physical representations of personal data in everyday life.
Her former training as an artist, in combination with her
background in design research inspires her to follow a
research through design approach. Her main research
interests revolve around data physicalization, personal
informatics, everyday objects and meaningfulness.
Danielle Wilde is Associate Professor of Embodied
Design at the University of Southern Denmark (SDU),
Kolding, where she leads SDU’s [body|bio] Soft Lab. She
specialises in participatory, speculative and critical research-
through-design, bringing focus to the social and ecological
impact of body-technology pairings and human-food
interactions. Core to her research, Wilde develops new
methods for thinking through moving, making and doing.
These methods enable diverse stakeholders to engage with
problems that cut cross disciplines and cultures, and develop
new practices, policies and technologies through a bottom-up
approach. Wilde publishes and exhibits widely across HCI,
Design Research, Citizen Science and Food Studies. 2020-
2021, she is Visiting Professor at Estonia Academy of Art
Doctoral School, helping them to build their research
capacity. For more, see: www.daniellewilde.com
Mette Gislev Kjærsgaard has worked with combinations
of design and anthropology in organizational as well as
academic contexts for the past twenty years. In 2011 she
received her Phd in Anthropology from the Department of
Culture and Society, University of Aarhus. She currently
holds a position as associate professor of design anthropology
at SDU design, University of Southern Denmark. Mette has
conducted research on i.a. community-based innovation,
digital play and design processes with a particular focus on
the applied and academic potentials of a design
anthropological approach. She is co-organizer of Research
Network for Design Anthropology and co-editor of Design
Anthropological Futures (Bloomsbury 2016).
Authorized licensed use limited to: Carleton University. Downloaded on November 01,2020 at 14:10:25 UTC from IEEE Xplore. Restrictions apply.